Skip to content

test: increasing unit test coverage to 95 percent - #740

Merged
GregHolmes merged 11 commits into
mainfrom
increase_unit_test_coverage
Aug 14, 2026
Merged

test: increasing unit test coverage to 95 percent#740
GregHolmes merged 11 commits into
mainfrom
increase_unit_test_coverage

Conversation

@mgama-deepgram

@mgama-deepgram mgama-deepgram commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Increase unit test coverage (~95%) + coverage in CI

Adds a hand-written custom test suite that raises deepgram package coverage to the ~95% target (branch coverage enabled), wires coverage reporting into CI with an automatic sticky PR comment, and scopes coverage to hand-maintainable logic rather than Fern-generated boilerplate. New tests and the coverage config are frozen in .fernignore so they survive the next SDK regeneration.

Coverage on the scoped source went from ~54% line / ~32% branch to 94.72% (gate fail_under = 90). No product code changed — this PR is tests + CI + config only.

What changed

  • CI (.github/workflows/ci.yml)

    • Runs on pull_request in addition to push (push now scoped to main + tags).
    • Test step runs under coverage: pytest -rP --cov=deepgram --cov-branch --cov-report=xml --cov-report=term-missing .
    • Generates a coverage summary and posts/updates a sticky PR comment (Python 3.13 leg only); also writes it to the job summary. Fork PRs are skipped for the comment (read-only token) and the step is best-effort.
    • Grants the test job pull-requests: write (with contents: read) so it can comment.
  • Coverage config (.coveragerc, not pyproject.toml)

    • [run] branch coverage on, source = deepgram, with pure-generated code excluded (types/, requests/, __init__.py, version.py, unused core/http_sse/) so the metric reflects code that carries logic.
    • [report] fail_under = 90, show_missing = True.
    • Kept out of pyproject.toml deliberately: freezing pyproject.toml would also block Fern's generator dependency updates.
  • Dev dependencies (pyproject.toml)

    • Adds pytest-cov and respx (HTTP mocking) to the dev group.
  • New tests (tests/custom/, ~170 test functions, many parametrized)

    • test_http_endpoints_coverage.py — table-driven sweep of REST endpoints (success + 400/403/non-JSON error branches, sync + async).
    • test_http_retry_coverage.py — retry/backoff behavior (exponential backoff, Retry-After, Retry-After-ms, X-RateLimit-Reset, exhaustion).
    • test_websocket_streaming_coverage.py — websocket streaming clients.
    • test_speak_v2_coverage.py — Speak V2 (Flux TTS): batch REST audio.generate streaming, connect header/handshake-failure branches, socket-client iteration/send/recv. Raises speak.v2 from 44% → 97%.
    • test_core_internals_branches.py, test_core_utilities_coverage.py, test_client_construction_and_helpers.py, test_logging_and_retry_branches.py — core utilities, client construction, logging/retry, and remaining branch gaps.
  • Regen safety / housekeeping

    • .fernignore — freezes the 8 new custom test files, .coveragerc, and .gitignore. Documented in AGENTS.md.
    • .gitignore — ignores .coverage, htmlcov/, coverage.xml (and the previously-tracked .coverage artifact is now untracked).
    • Rebased onto latest main (7.7.0 regen) to keep the branch current.

Testing

  • New tests use respx (HTTP) and in-process websocket fakes — no network / no WireMock dependency.
  • Full suite: 981 passed, 1 skipped under coverage locally; CI runs the full suite on Python 3.10–3.13.

Open item (for @greg / Fern pipeline): pyproject.toml is intentionally unfrozen, but now carries the respx / pytest-cov dev deps the frozen tests need. Confirm these survive the next Fern pyproject.toml regen (or move them into the generator's dependency config) so the frozen suite doesn't break on a future regen.

@mgama-deepgram mgama-deepgram changed the title Increase unit test coverage tests: Increase unit test coverage to 90 percent Jun 30, 2026
@mgama-deepgram
mgama-deepgram marked this pull request as ready for review June 30, 2026 17:00
@mgama-deepgram mgama-deepgram changed the title tests: Increase unit test coverage to 90 percent test: Increase unit test coverage to 90 percent Jun 30, 2026
@mgama-deepgram mgama-deepgram changed the title test: Increase unit test coverage to 90 percent test: increasing unit test coverage to 90 percent Jun 30, 2026
@mgama-deepgram mgama-deepgram changed the title test: increasing unit test coverage to 90 percent test: increasing unit test coverage to 95 percent Jul 2, 2026
@GregHolmes

Copy link
Copy Markdown
Contributor

Hi @mgama-deepgram this is great, however some of the files you've modified are generated files based on the API specs. I'd rather we didn't freeze files unless absolutely unavoidable.

@mgama-deepgram

Copy link
Copy Markdown
Contributor Author

@GregHolmes I removed all the changes that were done to files under src/deepgram/core and left only the new tests under tests/custom.
The code coverage is still really high, so that did not affect it that much.

@GregHolmes

Copy link
Copy Markdown
Contributor

@GregHolmes I removed all the changes that were done to files under src/deepgram/core and left only the new tests under tests/custom. The code coverage is still really high, so that did not affect it that much.

@mgama-deepgram would you not want your other test files in .fernignore? otherwise when regenerations done, it'll remove them.

@mgama-deepgram

Copy link
Copy Markdown
Contributor Author

@GregHolmes I removed all the changes that were done to files under src/deepgram/core and left only the new tests under tests/custom. The code coverage is still really high, so that did not affect it that much.

@mgama-deepgram would you not want your other test files in .fernignore? otherwise when regenerations done, it'll remove them.

Thanks for calling this out @GregHolmes I just added the tests files to the .fernignore file. I am new to use these type of files and regenerations, thanks for explaining.

@mgama-deepgram

Copy link
Copy Markdown
Contributor Author

@GregHolmes this is ready for another round of review. I added more test coverage for Speak V2 (Flux TTS). Thanks in advance

@mgama-deepgram
mgama-deepgram requested a review from jherl-dg July 28, 2026 19:59
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Branch Rate Complexity Health
src.deepgram 96% 92% 0
src.deepgram.agent 100% 100% 0
src.deepgram.agent.v1 98% 100% 0
src.deepgram.agent.v1.settings 100% 100% 0
src.deepgram.agent.v1.settings.think 100% 100% 0
src.deepgram.agent.v1.settings.think.models 97% 100% 0
src.deepgram.auth 100% 100% 0
src.deepgram.auth.v1 100% 100% 0
src.deepgram.auth.v1.tokens 97% 100% 0
src.deepgram.core 88% 81% 0
src.deepgram.errors 100% 100% 0
src.deepgram.helpers 100% 95% 0
src.deepgram.listen 100% 100% 0
src.deepgram.listen.v1 98% 93% 0
src.deepgram.listen.v1.media 97% 100% 0
src.deepgram.listen.v2 98% 92% 0
src.deepgram.manage 100% 100% 0
src.deepgram.manage.v1 100% 100% 0
src.deepgram.manage.v1.models 96% 100% 0
src.deepgram.manage.v1.projects 97% 100% 0
src.deepgram.manage.v1.projects.billing 100% 100% 0
src.deepgram.manage.v1.projects.billing.balances 96% 100% 0
src.deepgram.manage.v1.projects.billing.breakdown 97% 100% 0
src.deepgram.manage.v1.projects.billing.fields 97% 100% 0
src.deepgram.manage.v1.projects.billing.purchases 97% 100% 0
src.deepgram.manage.v1.projects.keys 96% 100% 0
src.deepgram.manage.v1.projects.members 97% 100% 0
src.deepgram.manage.v1.projects.members.invites 96% 100% 0
src.deepgram.manage.v1.projects.members.scopes 96% 100% 0
src.deepgram.manage.v1.projects.models 96% 100% 0
src.deepgram.manage.v1.projects.usage 98% 100% 0
src.deepgram.manage.v1.projects.usage.breakdown 97% 100% 0
src.deepgram.manage.v1.projects.usage.fields 97% 100% 0
src.deepgram.read 100% 100% 0
src.deepgram.read.v1 100% 100% 0
src.deepgram.read.v1.text 98% 100% 0
src.deepgram.self_hosted 100% 100% 0
src.deepgram.self_hosted.v1 100% 100% 0
src.deepgram.self_hosted.v1.distribution_credentials 96% 100% 0
src.deepgram.speak 100% 100% 0
src.deepgram.speak.v1 98% 97% 0
src.deepgram.speak.v1.audio 91% 80% 0
src.deepgram.speak.v2 98% 93% 0
src.deepgram.speak.v2.audio 100% 100% 0
src.deepgram.voice_agent 100% 100% 0
src.deepgram.voice_agent.configurations 95% 100% 0
src.deepgram.voice_agent.variables 95% 100% 0
Summary 95% (6478 / 6793) 91% (1410 / 1544) 0

Scope: hand-maintained SDK logic. Fern-generated data models (types/, requests/), package __init__.py files, version.py, and the unused core/http_sse/ scaffolding are excluded — see .coveragerc. Unscoped whole-package coverage is ~70%.

@mgama-deepgram

Copy link
Copy Markdown
Contributor Author

@dg-coreylweathers @jherl-dg this is ready for review. I appreciate your help, thanks in advance

@mgama-deepgram

Copy link
Copy Markdown
Contributor Author

@dg-coreylweathers @GregHolmes these changes are ready to be re-review. I have addressed the original changes requested from the first review. Thanks in advance

@GregHolmes
GregHolmes force-pushed the increase_unit_test_coverage branch from 45010dc to 0f3ed64 Compare August 14, 2026 14:59
Removes the committed .coverage binary from version control. It is a
local coverage artifact regenerated on every test run and is already
listed in .gitignore.
Pins irongut/CodeCoverageSummary (v1.3.0) and
marocchino/sticky-pull-request-comment (v2.9.4) to commit SHAs. Both
run with pull-requests: write, so pinning to an immutable SHA (rather
than a moving tag) hardens the privileged coverage-comment steps.
CodeCoverageSummary is a Docker container action that writes
code-coverage-results.md as root, so the follow-up annotate step's plain
`>>` append ran as the runner user and failed with 'Permission denied',
failing the 3.13 leg and cancelling the rest of the matrix (fail-fast).
Append as root via 'sudo tee -a' instead.
@GregHolmes
GregHolmes merged commit 921983a into main Aug 14, 2026
10 checks passed
@GregHolmes
GregHolmes deleted the increase_unit_test_coverage branch August 14, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants